Skip to content

Run pipelines test against the pipeline's testing.sources - #7429

Merged
tharunbhuk-db merged 7 commits into
mainfrom
tharun/pipelines-test-j1-testing-sources
Nov 18, 2026
Merged

tharunbhuk-db merged 7 commits into
mainfrom
tharun/pipelines-test-j1-testing-sources

Conversation

@tharunbhuk-db

@tharunbhuk-db tharunbhuk-db commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

🥞 Stacked PR


  • databricks pipelines test runs without an environment variable. Its help marks it Beta, so it stays unstable under the stability policy.
  • A run of all tests sends test_options as an empty object, so a workspace that does not support it returns a clear error.
  • The pipelines test acceptance bundles declare testing, since test runs need testing.sources.
  • When Start Update fails with PIPELINE_TEST_NO_TEST_SOURCES and the bundle sets testing, the error says to deploy first.
  • Adds the changelog entry for the command and the testing property.

Tests: TestNewStartTestUpdateRequest, TestWithDeployHint; acceptance pipelines/test/not-enabled, pipelines/test/no-test-sources, pipelines/databricks-cli-help, and the pipelines/test goldens without the gate cases.

This pull request and its description were written by Isaac.

@github-actions

github-actions Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Commit: 7ca3bea75 | Workflow run

Regression test report

Selected 8 changed tests (limit=20, 0 not selected)

8 tests: regression=8 unreleased=0 coverage=0 failing=0
test branch main (73a2b68) latest (v1.26.0)
TestAccept/pipelines/databricks-cli-help ✅ ❌ ❌
TestAccept/pipelines/test/arguments ✅ ❌ ❌
TestAccept/pipelines/test/immutable ✅ ❌ ❌
TestAccept/pipelines/test/no-test-sources ✅ ❌ ❌
TestAccept/pipelines/test/not-enabled ✅ ❌ ❌
TestAccept/pipelines/test/run-control/timeout ✅ ❌ ❌
TestAccept/pipelines/test/selection ✅ ❌ ❌
TestAccept/pipelines/test/submit-error ✅ ❌ ❌
TestAccept/pipelines/databricks-cli-help: branch ✅ | main (73a2b68) ❌ | latest (v1.26.0) ❌

main (73a2b68):

acceptance_test.go:1384: Diff:
        --- pipelines/databricks-cli-help/output.txt
        +++ /tmp/TestAcceptpipelinesdatabricks-cli-help1139207626/002/output.txt
        @@ -30,7 +30,7 @@
           run                   Run a pipeline
           show                  Preview a table's columns and sample rows
           stop                  Stop a pipeline
        -  test                  *Beta* Run the tests of a pipeline
        +  test                  *Experimental* Run the tests of a pipeline
         
         Management Commands
           apply-environment     *Public Preview* Apply the latest environment to the pipeline.

latest (v1.26.0):

acceptance_test.go:1384: Diff:
        --- pipelines/databricks-cli-help/output.txt
        +++ /tmp/TestAcceptpipelinesdatabricks-cli-help1593511623/002/output.txt
        @@ -30,7 +30,7 @@
           run                   Run a pipeline
           show                  Preview a table's columns and sample rows
           stop                  Stop a pipeline
        -  test                  *Beta* Run the tests of a pipeline
        +  test                  *Preview* Run the tests of a pipeline
         
         Management Commands
           apply-environment     *Public Preview* Apply the latest environment to the pipeline.
        @@ -52,13 +52,13 @@
           update-permissions    Update pipeline permissions.
         
         Flags:
        -  -h, --help[=true|false]   help for pipelines
        -      --var strings         set values for variables defined in project config. Example: --var="foo=bar"
        +  -h, --help          help for pipelines
        +      --var strings   set values for variables defined in project config. Example: --var="foo=bar"
         
         Global Flags:
        -      --debug[=true|false]   enable debug logging
        -  -o, --output type          output type: text or json (default text)
        -  -p, --profile string       ~/.databrickscfg profile
        -  -t, --target string        bundle target to use (if applicable)
        +      --debug            enable debug logging
        +  -o, --output type      output type: text or json (default text)
        +  -p, --profile string   ~/.databrickscfg profile
        +  -t, --target string    bundle target to use (if applicable)
         
         Use "databricks pipelines [command] --help" for more information about a command.
TestAccept/pipelines/test/arguments: branch ✅ | main (73a2b68) ❌ | latest (v1.26.0) ❌

main (73a2b68):

acceptance_test.go:1384: Diff:
        --- pipelines/test/arguments/output.txt
        +++ /tmp/TestAcceptpipelinestestarguments2363175690/002/output.txt
        @@ -2,6 +2,8 @@
         >>> [CLI] pipelines test --help
         Run the tests of a pipeline in a test-only update.
         
        +This command is experimental. Set DATABRICKS_CLI_PIPELINES_TEST=true to enable it.
        +
         KEY is the unique name of the pipeline, as defined in its YAML file.
         It is required when test selections are given. If there is only one pipeline
         in the project and no arguments are given, it is selected automatically.
        @@ -62,52 +64,52 @@
         Error: -k is a pytest argument: pass pytest arguments with --pytest-arg, for example --pytest-arg=-k --pytest-arg='orders and not slow'
         
         >>> musterr [CLI] pipelines test --pipeline-id 1234 -- -k orders
        -Error: the -- separator is not supported: pass pytest arguments with --pytest-arg, for example --pytest-arg=-k --pytest-arg='orders and not slow'
        +Error: pipelines test is experimental. Set DATABRICKS_CLI_PIPELINES_TEST=true to enable it
         
         === Invalid --pytest-arg values fail before any request
         
         >>> musterr [CLI] pipelines test --pipeline-id 1234 --pytest-arg=-x
        -Error: unsupported pytest argument "-x": only -k and -m are supported
        +Error: pipelines test is experimental. Set DATABRICKS_CLI_PIPELINES_TEST=true to enable it
         
         >>> musterr [CLI] pipelines test --pipeline-id 1234 --pytest-arg=
        -Error: --pytest-arg requires a non-empty value
        +Error: pipelines test is experimental. Set DATABRICKS_CLI_PIPELINES_TEST=true to enable it
         
         === The pipeline ID must be a pipeline ID
         
         >>> musterr [CLI] pipelines test --pipeline-id 1234
        -Error: --pipeline-id "1234" is not a pipeline ID
        +Error: pipelines test is experimental. Set DATABRICKS_CLI_PIPELINES_TEST=true to enable it
         
         === JSON output is not supported
         
         >>> musterr [CLI] pipelines test --pipeline-id 1234 -o json
        -Error: --output json is not supported by pipelines test
        +Error: pipelines test is experimental. Set DATABRICKS_CLI_PIPELINES_TEST=true to enable it
         
         === Incompatible run-control flags
         
         >>> musterr [CLI] pipelines test --pipeline-id 1234 tests --resume 5678
        -Error: --resume cannot be used with test selections
        +Error: pipelines test is experimental. Set DATABRICKS_CLI_PIPELINES_TEST=true to enable it
         
         >>> musterr [CLI] pipelines test --pipeline-id 1234 --resume 5678 --pytest-arg=-m --pytest-arg=smoke
        -Error: --resume cannot be used with --pytest-arg
        +Error: pipelines test is experimental. Set DATABRICKS_CLI_PIPELINES_TEST=true to enable it
         
         >>> musterr [CLI] pipelines test --pipeline-id 1234 --resume 5678 --no-wait
        -Error: --resume cannot be used with --no-wait
        +Error: pipelines test is experimental. Set DATABRICKS_CLI_PIPELINES_TEST=true to enable it
         
         >>> musterr [CLI] pipelines test --pipeline-id 1234 --no-wait --timeout 10m
        -Error: --timeout cannot be used with --no-wait
        +Error: pipelines test is experimental. Set DATABRICKS_CLI_PIPELINES_TEST=true to enable it
         
         >>> musterr [CLI] pipelines test --pipeline-id 1234 --resume 5678' OR '1'='1
        -Error: --resume "5678' OR '1'='1" is not an update ID; pass the update ID that pipelines test printed
        +Error: pipelines test is experimental. Set DATABRICKS_CLI_PIPELINES_TEST=true to enable it
         
         >>> musterr [CLI] pipelines test --pipeline-id 1234 --resume 
        -Error: --resume "" is not an update ID; pass the update ID that pipelines test printed
        +Error: pipelines test is experimental. Set DATABRICKS_CLI_PIPELINES_TEST=true to enable it
         
         === --junit-xml needs the results
         
         >>> musterr [CLI] pipelines test --pipeline-id 1234 --no-wait --junit-xml out.no-wait.xml
        -Error: --junit-xml cannot be used with --no-wait because no results are available when the command returns
        +Error: pipelines test is experimental. Set DATABRICKS_CLI_PIPELINES_TEST=true to enable it
         
         === An invalid flag replaces an older report
         
         >>> musterr [CLI] pipelines test --pipeline-id 1234 --pytest-arg=-x --junit-xml out.flag-error.xml
        -Error: unsupported pytest argument "-x": only -k and -m are supported
        +Error: pipelines test is experimental. Set DATABRICKS_CLI_PIPELINES_TEST=true to enable it
        
    acceptance_test.go:1384: 
        	Error Trace:	/home/runner/work/cli/cli/libs/testdiff/testdiff.go:22
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1384
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1137
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:670
        	Error:      	Not equal: 
        	            	expected: "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<testsuites>\n  <testsuite name=\"1234\" tests=\"1\" failures=\"0\" errors=\"1\" skipped=\"0\">\n    <testcase name=\"pipelines test\">\n      <error message=\"unsupported pytest argument &#34;-x&#34;: only -k and -m are supported\">unsupported pytest argument &#34;-x&#34;: only -k and -m are supported</error>\n    </testcase>\n  </testsuite>\n</testsuites>\n"
        	            	actual  : "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<testsuites>\n  <testsuite name=\"1234\" tests=\"1\" failures=\"0\" errors=\"1\" skipped=\"0\">\n    <testcase name=\"pipelines test\">\n      <error message=\"pipelines test is experimental. Set DATABRICKS_CLI_PIPELINES_TEST=true to enable it\">pipelines test is experimental. Set DATABRICKS_CLI_PIPELINES_TEST=true to enable it</error>\n    </testcase>\n  </testsuite>\n</testsuites>\n"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -4,3 +4,3 @@
        	            	     <testcase name="pipelines test">
        	            	-      <error message="unsupported pytest argument &#34;-x&#34;: only -k and -m are supported">unsupported pytest argument &#34;-x&#34;: only -k and -m are supported</error>
        	            	+      <error message="pipelines test is experimental. Set DATABRICKS_CLI_PIPELINES_TEST=true to enable it">pipelines test is experimental. Set DATABRICKS_CLI_PIPELINES_TEST=true to enable it</error>
        	            	     </testcase>
        	Test:       	TestAccept/pipelines/test/arguments
        	Messages:   	pipelines/test/arguments/out.flag-error.xml vs /tmp/TestAcceptpipelinestestarguments2363175690/001/out.flag-error.xml
    acceptance_test.go:1384: 
        	Error Trace:	/home/runner/work/cli/cli/libs/testdiff/testdiff.go:22
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1384
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1137
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:670
        	Error:      	Not equal: 
        	            	expected: "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<testsuites>\n  <testsuite name=\"1234\" tests=\"1\" failures=\"0\" errors=\"1\" skipped=\"0\">\n    <testcase name=\"pipelines test\">\n      <error message=\"--junit-xml cannot be used with --no-wait because no results are available when the command returns\">--junit-xml cannot be used with --no-wait because no results are available when the command returns</error>\n    </testcase>\n  </testsuite>\n</testsuites>\n"
        	            	actual  : "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<testsuites>\n  <testsuite name=\"1234\" tests=\"1\" failures=\"0\" errors=\"1\" skipped=\"0\">\n    <testcase name=\"pipelines test\">\n      <error message=\"pipelines test is experimental. Set DATABRICKS_CLI_PIPELINES_TEST=true to enable it\">pipelines test is experimental. Set DATABRICKS_CLI_PIPELINES_TEST=true to enable it</error>\n    </testcase>\n  </testsuite>\n</testsuites>\n"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -4,3 +4,3 @@
        	            	     <testcase name="pipelines test">
        	            	-      <error message="--junit-xml cannot be used with --no-wait because no results are available when the command returns">--junit-xml cannot be used with --no-wait because no results are available when the command returns</error>
        	            	+      <error message="pipelines test is experimental. Set DATABRICKS_CLI_PIPELINES_TEST=true to enable it">pipelines test is experimental. Set DATABRICKS_CLI_PIPELINES_TEST=true to enable it</error>
        	            	     </testcase>
        	Test:       	TestAccept/pipelines/test/arguments
        	Messages:   	pipelines/test/arguments/out.no-wait.xml vs /tmp/TestAcceptpipelinestestarguments2363175690/001/out.no-wait.xml

latest (v1.26.0):

acceptance_test.go:1384: Diff:
        --- pipelines/test/arguments/output.txt
        +++ /tmp/TestAcceptpipelinestestarguments4129819716/002/output.txt
        @@ -2,6 +2,8 @@
         >>> [CLI] pipelines test --help
         Run the tests of a pipeline in a test-only update.
         
        +This command is experimental. Set DATABRICKS_CLI_PIPELINES_TEST=true to enable it.
        +
         KEY is the unique name of the pipeline, as defined in its YAML file.
         It is required when test selections are given. If there is only one pipeline
         in the project and no arguments are given, it is selected automatically.
        @@ -21,40 +23,22 @@
         
           databricks pipelines test my_pipeline --pytest-arg=-k --pytest-arg='orders and not slow'
         
        -The command waits for the results. It exits with code 0 only if at least one
        -test was collected and none failed or errored, so a run whose tests were all
        -skipped succeeds; otherwise it exits with code 1. --no-wait returns after the
        -update starts.
        -
        ---timeout and Ctrl+C stop waiting but not the update; pass its ID to --resume
        -to read the results later. To stop the update, run 'pipelines stop'.
        -
        ---junit-xml writes one entry per test with its outcome and duration. An
        -incomplete run, a run without tests, or a timeout adds a failing entry that
        -names the cause. The report replaces the file on every exit except a flag that
        -cannot be parsed. With workspace.sync_mode immutable, test names keep the path
        -of the deployed snapshot, which changes with each deployment.
        -
         Usage:
           databricks pipelines test [flags] [KEY] [TEST_SELECTION...]
         
         Pipeline Test Flags:
        -      --junit-xml string         Write the results as a JUnit XML report to this file, replacing it.
        -      --no-wait[=true|false]     Don't wait for the tests to complete.
               --pipeline-id string       ID of the pipeline to test, instead of a KEY.
               --pytest-arg stringArray   Argument to pass to pytest (repeatable). Only -k and -m are supported.
        -      --resume string            Read the results of an existing test update instead of starting one.
        -      --timeout duration         Stop waiting for results after this duration, for example 30m. The update keeps running.
         
         Flags:
        -  -h, --help[=true|false]   help for test
        +  -h, --help   help for test
         
         Global Flags:
        -      --debug[=true|false]   enable debug logging
        -  -o, --output type          output type: text or json (default text)
        -  -p, --profile string       ~/.databrickscfg profile
        -  -t, --target string        bundle target to use (if applicable)
        -      --var strings          set values for variables defined in project config. Example: --var="foo=bar"
        +      --debug            enable debug logging
        +  -o, --output type      output type: text or json (default text)
        +  -p, --profile string   ~/.databrickscfg profile
        +  -t, --target string    bundle target to use (if applicable)
        +      --var strings      set values for variables defined in project config. Example: --var="foo=bar"
         
         === pytest arguments must be passed with --pytest-arg
         
        @@ -62,52 +36,196 @@
         Error: -k is a pytest argument: pass pytest arguments with --pytest-arg, for example --pytest-arg=-k --pytest-arg='orders and not slow'
         
         >>> musterr [CLI] pipelines test --pipeline-id 1234 -- -k orders
        -Error: the -- separator is not supported: pass pytest arguments with --pytest-arg, for example --pytest-arg=-k --pytest-arg='orders and not slow'
        +Error: pipelines test is experimental. Set DATABRICKS_CLI_PIPELINES_TEST=true to enable it
         
         === Invalid --pytest-arg values fail before any request
         
         >>> musterr [CLI] pipelines test --pipeline-id 1234 --pytest-arg=-x
        -Error: unsupported pytest argument "-x": only -k and -m are supported
        +Error: pipelines test is experimental. Set DATABRICKS_CLI_PIPELINES_TEST=true to enable it
         
         >>> musterr [CLI] pipelines test --pipeline-id 1234 --pytest-arg=
        -Error: --pytest-arg requires a non-empty value
        +Error: pipelines test is experimental. Set DATABRICKS_CLI_PIPELINES_TEST=true to enable it
         
         === The pipeline ID must be a pipeline ID
         
         >>> musterr [CLI] pipelines test --pipeline-id 1234
        -Error: --pipeline-id "1234" is not a pipeline ID
        +Error: pipelines test is experimental. Set DATABRICKS_CLI_PIPELINES_TEST=true to enable it
         
         === JSON output is not supported
         
         >>> musterr [CLI] pipelines test --pipeline-id 1234 -o json
        -Error: --output json is not supported by pipelines test
        +Error: pipelines test is experimental. Set DATABRICKS_CLI_PIPELINES_TEST=true to enable it
         
         === Incompatible run-control flags
         
         >>> musterr [CLI] pipelines test --pipeline-id 1234 tests --resume 5678
        -Error: --resume cannot be used with test selections
        +Error: unknown flag: --resume
         
        +Usage:
        +  databricks pipelines test [flags] [KEY] [TEST_SELECTION...]
        +
        +Pipeline Test Flags:
        +      --pipeline-id string       ID of the pipeline to test, instead of a KEY.
        +      --pytest-arg stringArray   Argument to pass to pytest (repeatable). Only -k and -m are supported.
        +
        +Flags:
        +  -h, --help   help for test
        +
        +Global Flags:
        +      --debug            enable debug logging
        +  -o, --output type      output type: text or json (default text)
        +  -p, --profile string   ~/.databrickscfg profile
        +  -t, --target string    bundle target to use (if applicable)
        +      --var strings      set values for variables defined in project config. Example: --var="foo=bar"
        +
        +
         >>> musterr [CLI] pipelines test --pipeline-id 1234 --resume 5678 --pytest-arg=-m --pytest-arg=smoke
        -Error: --resume cannot be used with --pytest-arg
        +Error: unknown flag: --resume
         
        +Usage:
        +  databricks pipelines test [flags] [KEY] [TEST_SELECTION...]
        +
        +Pipeline Test Flags:
        +      --pipeline-id string       ID of the pipeline to test, instead of a KEY.
        +      --pytest-arg stringArray   Argument to pass to pytest (repeatable). Only -k and -m are supported.
        +
        +Flags:
        +  -h, --help   help for test
        +
        +Global Flags:
        +      --debug            enable debug logging
        +  -o, --output type      output type: text or json (default text)
        +  -p, --profile string   ~/.databrickscfg profile
        +  -t, --target string    bundle target to use (if applicable)
        +      --var strings      set values for variables defined in project config. Example: --var="foo=bar"
        +
        +
         >>> musterr [CLI] pipelines test --pipeline-id 1234 --resume 5678 --no-wait
        -Error: --resume cannot be used with --no-wait
        +Error: unknown flag: --resume
         
        +Usage:
        +  databricks pipelines test [flags] [KEY] [TEST_SELECTION...]
        +
        +Pipeline Test Flags:
        +      --pipeline-id string       ID of the pipeline to test, instead of a KEY.
        +      --pytest-arg stringArray   Argument to pass to pytest (repeatable). Only -k and -m are supported.
        +
        +Flags:
        +  -h, --help   help for test
        +
        +Global Flags:
        +      --debug            enable debug logging
        +  -o, --output type      output type: text or json (default text)
        +  -p, --profile string   ~/.databrickscfg profile
        +  -t, --target string    bundle target to use (if applicable)
        +      --var strings      set values for variables defined in project config. Example: --var="foo=bar"
        +
        +
         >>> musterr [CLI] pipelines test --pipeline-id 1234 --no-wait --timeout 10m
        -Error: --timeout cannot be used with --no-wait
        +Error: unknown flag: --no-wait
         
        +Usage:
        +  databricks pipelines test [flags] [KEY] [TEST_SELECTION...]
        +
        +Pipeline Test Flags:
        +      --pipeline-id string       ID of the pipeline to test, instead of a KEY.
        +      --pytest-arg stringArray   Argument to pass to pytest (repeatable). Only -k and -m are supported.
        +
        +Flags:
        +  -h, --help   help for test
        +
        +Global Flags:
        +      --debug            enable debug logging
        +  -o, --output type      output type: text or json (default text)
        +  -p, --profile string   ~/.databrickscfg profile
        +  -t, --target string    bundle target to use (if applicable)
        +      --var strings      set values for variables defined in project config. Example: --var="foo=bar"
        +
        +
         >>> musterr [CLI] pipelines test --pipeline-id 1234 --resume 5678' OR '1'='1
        -Error: --resume "5678' OR '1'='1" is not an update ID; pass the update ID that pipelines test printed
        +Error: unknown flag: --resume
         
        +Usage:
        +  databricks pipelines test [flags] [KEY] [TEST_SELECTION...]
        +
        +Pipeline Test Flags:
        +      --pipeline-id string       ID of the pipeline to test, instead of a KEY.
        +      --pytest-arg stringArray   Argument to pass to pytest (repeatable). Only -k and -m are supported.
        +
        +Flags:
        +  -h, --help   help for test
        +
        +Global Flags:
        +      --debug            enable debug logging
        +  -o, --output type      output type: text or json (default text)
        +  -p, --profile string   ~/.databrickscfg profile
        +  -t, --target string    bundle target to use (if applicable)
        +      --var strings      set values for variables defined in project config. Example: --var="foo=bar"
        +
        +
... (96 more lines truncated, raise DATABRICKS_CLI_TEST_MAX_LOG to see more)
TestAccept/pipelines/test/immutable: branch ✅ | main (73a2b68) ❌ | latest (v1.26.0) ❌

main (73a2b68):

acceptance_test.go:1384: Diff:
        --- pipelines/test/immutable/output.txt
        +++ /tmp/TestAcceptpipelinestestimmutable3415898093/002/output.txt
        @@ -4,53 +4,16 @@
         Created pipelines.my_pipeline
         Files: 0 uploaded, 0 deleted
         Resources: 2 created, 0 changed, 0 deleted, 0 unchanged
        -View your pipeline my_pipeline here: [DATABRICKS_URL]/pipelines/[UUID][0]?w=[WORKSPACE_ID]
        +View your pipeline my_pipeline here: [DATABRICKS_URL]/pipelines/[UUID]?w=[WORKSPACE_ID]
         
         === Relative test paths are rejected with workspace.sync_mode immutable
         
         >>> musterr [CLI] pipelines test my_pipeline tests/test_orders.py::test_backfill
        -Recommendation: This command runs the last deployed version of the code
        +Error: pipelines test is experimental. Set DATABRICKS_CLI_PIPELINES_TEST=true to enable it
         
        -If you've made local changes, run 'databricks pipelines deploy' first to ensure they are included.
        -
        -Error: cannot resolve test selection "tests/test_orders.py::test_backfill": relative test paths need synced local files, but workspace.sync_mode is immutable; pass the /Workspace path
        -
         === Absolute workspace paths and running all tests still work
         
         >>> MSYS_NO_PATHCONV=1 [CLI] pipelines test my_pipeline /Workspace/Users/someone@example.com/tests/test_orders.py::test_backfill --no-wait
        -Recommendation: This command runs the last deployed version of the code
        +Error: pipelines test is experimental. Set DATABRICKS_CLI_PIPELINES_TEST=true to enable it
         
        -If you've made local changes, run 'databricks pipelines deploy' first to ensure they are included.
        -
        -Pipeline: my_pipeline ([UUID][0])
        -Update: [UUID][1]
        -
        ->>> [CLI] pipelines test my_pipeline --no-wait
        -Recommendation: This command runs the last deployed version of the code
        -
        -If you've made local changes, run 'databricks pipelines deploy' first to ensure they are included.
        -
        -Pipeline: my_pipeline ([UUID][0])
        -Update: [UUID][2]
        -
        ->>> print_requests.py //updates
        -{
        -  "method": "POST",
        -  "path": "/api/2.0/pipelines/[UUID][0]/updates",
        -  "body": {
        -    "test_only": true,
        -    "test_options": {
        -      "test_selection": [
        -        "/Workspace/Users/someone@example.com/tests/test_orders.py::test_backfill"
        -      ]
        -    }
        -  }
        -}
        -{
        -  "method": "POST",
        -  "path": "/api/2.0/pipelines/[UUID][0]/updates",
        -  "body": {
        -    "test_only": true,
        -    "test_options": {}
        -  }
        -}
        +Exit code: 1

latest (v1.26.0):

acceptance_test.go:1384: Diff:
        --- pipelines/test/immutable/output.txt
        +++ /tmp/TestAcceptpipelinestestimmutable3289270457/002/output.txt
        @@ -1,56 +1,41 @@
         
         >>> [CLI] pipelines deploy
        +Warning: unknown field: testing
        +  at resources.pipelines.my_pipeline
        +  in databricks.yml:14:7
        +
         Created internal_immutable_snapshots.immutable
         Created pipelines.my_pipeline
         Files: 0 uploaded, 0 deleted
         Resources: 2 created, 0 changed, 0 deleted, 0 unchanged
        -View your pipeline my_pipeline here: [DATABRICKS_URL]/pipelines/[UUID][0]?w=[WORKSPACE_ID]
        +View your pipeline my_pipeline here: [DATABRICKS_URL]/pipelines/[UUID]?w=[WORKSPACE_ID]
         
         === Relative test paths are rejected with workspace.sync_mode immutable
         
         >>> musterr [CLI] pipelines test my_pipeline tests/test_orders.py::test_backfill
        -Recommendation: This command runs the last deployed version of the code
        +Error: pipelines test is experimental. Set DATABRICKS_CLI_PIPELINES_TEST=true to enable it
         
        -If you've made local changes, run 'databricks pipelines deploy' first to ensure they are included.
        -
        -Error: cannot resolve test selection "tests/test_orders.py::test_backfill": relative test paths need synced local files, but workspace.sync_mode is immutable; pass the /Workspace path
        -
         === Absolute workspace paths and running all tests still work
         
         >>> MSYS_NO_PATHCONV=1 [CLI] pipelines test my_pipeline /Workspace/Users/someone@example.com/tests/test_orders.py::test_backfill --no-wait
        -Recommendation: This command runs the last deployed version of the code
        +Error: unknown flag: --no-wait
         
        -If you've made local changes, run 'databricks pipelines deploy' first to ensure they are included.
        +Usage:
        +  databricks pipelines test [flags] [KEY] [TEST_SELECTION...]
         
        -Pipeline: my_pipeline ([UUID][0])
        -Update: [UUID][1]
        +Pipeline Test Flags:
        +      --pipeline-id string       ID of the pipeline to test, instead of a KEY.
        +      --pytest-arg stringArray   Argument to pass to pytest (repeatable). Only -k and -m are supported.
         
        ->>> [CLI] pipelines test my_pipeline --no-wait
        -Recommendation: This command runs the last deployed version of the code
        +Flags:
        +  -h, --help   help for test
         
        -If you've made local changes, run 'databricks pipelines deploy' first to ensure they are included.
        +Global Flags:
        +      --debug            enable debug logging
        +  -o, --output type      output type: text or json (default text)
        +  -p, --profile string   ~/.databrickscfg profile
        +  -t, --target string    bundle target to use (if applicable)
        +      --var strings      set values for variables defined in project config. Example: --var="foo=bar"
         
        -Pipeline: my_pipeline ([UUID][0])
        -Update: [UUID][2]
         
        ->>> print_requests.py //updates
        -{
        -  "method": "POST",
        -  "path": "/api/2.0/pipelines/[UUID][0]/updates",
        -  "body": {
        -    "test_only": true,
        -    "test_options": {
        -      "test_selection": [
        -        "/Workspace/Users/someone@example.com/tests/test_orders.py::test_backfill"
        -      ]
        -    }
        -  }
        -}
        -{
        -  "method": "POST",
        -  "path": "/api/2.0/pipelines/[UUID][0]/updates",
        -  "body": {
        -    "test_only": true,
        -    "test_options": {}
        -  }
        -}
        +Exit code: 1
TestAccept/pipelines/test/no-test-sources: branch ✅ | main (73a2b68) ❌ | latest (v1.26.0) ❌

main (73a2b68):

acceptance_test.go:1384: 
        	Error Trace:	/home/runner/work/cli/cli/libs/testdiff/testdiff.go:22
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1384
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1125
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:670
        	Error:      	Not equal: 
        	            	expected: "\n>>> [CLI] pipelines deploy\nUploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-pipeline-test-no-test-sources/default/files...\nCreated pipelines.my_pipeline\nFiles: 3 uploaded, 0 deleted\nResources: 1 created, 0 changed, 0 deleted, 0 unchanged\nView your pipeline my_pipeline here: [DATABRICKS_URL]/pipelines/[UUID]?w=[WORKSPACE_ID]\n\n=== The bundle now sets testing, so the service error comes with a hint to deploy\n\n>>> musterr [CLI] pipelines test --no-wait\nRecommendation: This command runs the last deployed version of the code\n\nIf you've made local changes, run 'databricks pipelines deploy' first to ensure they are included.\n\nError: [PIPELINE_TEST_NO_TEST_SOURCES] The pipeline has no test sources. Set `testing.sources` to the test files or folders to run.\nThe bundle sets testing for my_pipeline, but the deployed pipeline has none. Run 'databricks pipelines deploy' first\n"
        	            	actual  : "\n>>> [CLI] pipelines deploy\nUploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-pipeline-test-no-test-sources/default/files...\nCreated pipelines.my_pipeline\nFiles: 3 uploaded, 0 deleted\nResources: 1 created, 0 changed, 0 deleted, 0 unchanged\nView your pipeline my_pipeline here: [DATABRICKS_URL]/pipelines/[UUID]?w=[WORKSPACE_ID]\n\n=== The bundle now sets testing, so the service error comes with a hint to deploy\n\n>>> musterr [CLI] pipelines test --no-wait\nError: pipelines test is experimental. Set DATABRICKS_CLI_PIPELINES_TEST=true to enable it\n"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -11,8 +11,3 @@
        	            	 >>> musterr [CLI] pipelines test --no-wait
        	            	-Recommendation: This command runs the last deployed version of the code
        	            	+Error: pipelines test is experimental. Set DATABRICKS_CLI_PIPELINES_TEST=true to enable it
        	            	 
        	            	-If you've made local changes, run 'databricks pipelines deploy' first to ensure they are included.
        	            	-
        	            	-Error: [PIPELINE_TEST_NO_TEST_SOURCES] The pipeline has no test sources. Set `testing.sources` to the test files or folders to run.
        	            	-The bundle sets testing for my_pipeline, but the deployed pipeline has none. Run 'databricks pipelines deploy' first
        	            	-
        	Test:       	TestAccept/pipelines/test/no-test-sources
        	Messages:   	pipelines/test/no-test-sources/output.txt vs /tmp/TestAcceptpipelinestestno-test-sources2719890357/002/output.txt

latest (v1.26.0):

acceptance_test.go:1384: Diff:
        --- pipelines/test/no-test-sources/output.txt
        +++ /tmp/TestAcceptpipelinestestno-test-sources1469165618/002/output.txt
        @@ -9,9 +9,22 @@
         === The bundle now sets testing, so the service error comes with a hint to deploy
         
         >>> musterr [CLI] pipelines test --no-wait
        -Recommendation: This command runs the last deployed version of the code
        +Error: unknown flag: --no-wait
         
        -If you've made local changes, run 'databricks pipelines deploy' first to ensure they are included.
        +Usage:
        +  databricks pipelines test [flags] [KEY] [TEST_SELECTION...]
         
        -Error: [PIPELINE_TEST_NO_TEST_SOURCES] The pipeline has no test sources. Set `testing.sources` to the test files or folders to run.
        -The bundle sets testing for my_pipeline, but the deployed pipeline has none. Run 'databricks pipelines deploy' first
        +Pipeline Test Flags:
        +      --pipeline-id string       ID of the pipeline to test, instead of a KEY.
        +      --pytest-arg stringArray   Argument to pass to pytest (repeatable). Only -k and -m are supported.
        +
        +Flags:
        +  -h, --help   help for test
        +
        +Global Flags:
        +      --debug            enable debug logging
        +  -o, --output type      output type: text or json (default text)
        +  -p, --profile string   ~/.databrickscfg profile
        +  -t, --target string    bundle target to use (if applicable)
        +      --var strings      set values for variables defined in project config. Example: --var="foo=bar"
        +
TestAccept/pipelines/test/not-enabled: branch ✅ | main (73a2b68) ❌ | latest (v1.26.0) ❌

main (73a2b68):

acceptance_test.go:1384: 
        	Error Trace:	/home/runner/work/cli/cli/libs/testdiff/testdiff.go:22
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1384
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1125
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:670
        	Error:      	Not equal: 
        	            	expected: "\n=== Where the testing API is off, a run of all tests is rejected, not run as an older test-only update\n\n>>> musterr [CLI] pipelines test --pipeline-id [UUID] --junit-xml out.not-enabled.xml\nError: Pipeline testing is not enabled for this pipeline. Contact your Databricks account team to join the preview.\n\n>>> print_requests.py //updates\n{\n  \"method\": \"POST\",\n  \"path\": \"/api/2.0/pipelines/[UUID]/updates\",\n  \"body\": {\n    \"test_only\": true,\n    \"test_options\": {}\n  }\n}\n"
        	            	actual  : "\n=== Where the testing API is off, a run of all tests is rejected, not run as an older test-only update\n\n>>> musterr [CLI] pipelines test --pipeline-id [UUID] --junit-xml out.not-enabled.xml\nError: pipelines test is experimental. Set DATABRICKS_CLI_PIPELINES_TEST=true to enable it\n\n>>> print_requests.py //updates\nFile [OUTPUT_DIR]/out.requests.txt not found\n\nExit code: 1\n"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -4,13 +4,8 @@
        	            	 >>> musterr [CLI] pipelines test --pipeline-id [UUID] --junit-xml out.not-enabled.xml
        	            	-Error: Pipeline testing is not enabled for this pipeline. Contact your Databricks account team to join the preview.
        	            	+Error: pipelines test is experimental. Set DATABRICKS_CLI_PIPELINES_TEST=true to enable it
        	            	 
        	            	 >>> print_requests.py //updates
        	            	-{
        	            	-  "method": "POST",
        	            	-  "path": "/api/2.0/pipelines/[UUID]/updates",
        	            	-  "body": {
        	            	-    "test_only": true,
        	            	-    "test_options": {}
        	            	-  }
        	            	-}
        	            	+File [OUTPUT_DIR]/out.requests.txt not found
        	            	 
        	            	+Exit code: 1
        	            	+
        	Test:       	TestAccept/pipelines/test/not-enabled
        	Messages:   	pipelines/test/not-enabled/output.txt vs /tmp/TestAcceptpipelinestestnot-enabled3268619313/002/output.txt
    acceptance_test.go:1384: 
        	Error Trace:	/home/runner/work/cli/cli/libs/testdiff/testdiff.go:22
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1384
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1137
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:670
        	Error:      	Not equal: 
        	            	expected: "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<testsuites>\n  <testsuite name=\"[UUID]\" tests=\"1\" failures=\"0\" errors=\"1\" skipped=\"0\">\n    <testcase name=\"pipelines test\">\n      <error message=\"Pipeline testing is not enabled for this pipeline. Contact your Databricks account team to join the preview.\">Pipeline testing is not enabled for this pipeline. Contact your Databricks account team to join the preview.</error>\n    </testcase>\n  </testsuite>\n</testsuites>\n"
        	            	actual  : "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<testsuites>\n  <testsuite name=\"[UUID]\" tests=\"1\" failures=\"0\" errors=\"1\" skipped=\"0\">\n    <testcase name=\"pipelines test\">\n      <error message=\"pipelines test is experimental. Set DATABRICKS_CLI_PIPELINES_TEST=true to enable it\">pipelines test is experimental. Set DATABRICKS_CLI_PIPELINES_TEST=true to enable it</error>\n    </testcase>\n  </testsuite>\n</testsuites>\n"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -4,3 +4,3 @@
        	            	     <testcase name="pipelines test">
        	            	-      <error message="Pipeline testing is not enabled for this pipeline. Contact your Databricks account team to join the preview.">Pipeline testing is not enabled for this pipeline. Contact your Databricks account team to join the preview.</error>
        	            	+      <error message="pipelines test is experimental. Set DATABRICKS_CLI_PIPELINES_TEST=true to enable it">pipelines test is experimental. Set DATABRICKS_CLI_PIPELINES_TEST=true to enable it</error>
        	            	     </testcase>
        	Test:       	TestAccept/pipelines/test/not-enabled
        	Messages:   	pipelines/test/not-enabled/out.not-enabled.xml vs /tmp/TestAcceptpipelinestestnot-enabled3268619313/001/out.not-enabled.xml

latest (v1.26.0):

acceptance_test.go:1384: 
        	Error Trace:	/home/runner/work/cli/cli/libs/testdiff/testdiff.go:22
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1384
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1125
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:670
        	Error:      	Not equal: 
        	            	expected: "\n=== Where the testing API is off, a run of all tests is rejected, not run as an older test-only update\n\n>>> musterr [CLI] pipelines test --pipeline-id [UUID] --junit-xml out.not-enabled.xml\nError: Pipeline testing is not enabled for this pipeline. Contact your Databricks account team to join the preview.\n\n>>> print_requests.py //updates\n{\n  \"method\": \"POST\",\n  \"path\": \"/api/2.0/pipelines/[UUID]/updates\",\n  \"body\": {\n    \"test_only\": true,\n    \"test_options\": {}\n  }\n}\n"
        	            	actual  : "\n=== Where the testing API is off, a run of all tests is rejected, not run as an older test-only update\n\n>>> musterr [CLI] pipelines test --pipeline-id [UUID] --junit-xml out.not-enabled.xml\nError: unknown flag: --junit-xml\n\nUsage:\n  databricks pipelines test [flags] [KEY] [TEST_SELECTION...]\n\nPipeline Test Flags:\n      --pipeline-id string       ID of the pipeline to test, instead of a KEY.\n      --pytest-arg stringArray   Argument to pass to pytest (repeatable). Only -k and -m are supported.\n\nFlags:\n  -h, --help   help for test\n\nGlobal Flags:\n      --debug            enable debug logging\n  -o, --output type      output type: text or json (default text)\n  -p, --profile string   ~/.databrickscfg profile\n  -t, --target string    bundle target to use (if applicable)\n      --var strings      set values for variables defined in project config. Example: --var=\"foo=bar\"\n\n\n>>> print_requests.py //updates\nFile [OUTPUT_DIR]/out.requests.txt not found\n\nExit code: 1\n"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -4,13 +4,26 @@
        	            	 >>> musterr [CLI] pipelines test --pipeline-id [UUID] --junit-xml out.not-enabled.xml
        	            	-Error: Pipeline testing is not enabled for this pipeline. Contact your Databricks account team to join the preview.
        	            	+Error: unknown flag: --junit-xml
        	            	+
        	            	+Usage:
        	            	+  databricks pipelines test [flags] [KEY] [TEST_SELECTION...]
        	            	+
        	            	+Pipeline Test Flags:
        	            	+      --pipeline-id string       ID of the pipeline to test, instead of a KEY.
        	            	+      --pytest-arg stringArray   Argument to pass to pytest (repeatable). Only -k and -m are supported.
        	            	+
        	            	+Flags:
        	            	+  -h, --help   help for test
        	            	+
        	            	+Global Flags:
        	            	+      --debug            enable debug logging
        	            	+  -o, --output type      output type: text or json (default text)
        	            	+  -p, --profile string   ~/.databrickscfg profile
        	            	+  -t, --target string    bundle target to use (if applicable)
        	            	+      --var strings      set values for variables defined in project config. Example: --var="foo=bar"
        	            	+
        	            	 
        	            	 >>> print_requests.py //updates
        	            	-{
        	            	-  "method": "POST",
        	            	-  "path": "/api/2.0/pipelines/[UUID]/updates",
        	            	-  "body": {
        	            	-    "test_only": true,
        	            	-    "test_options": {}
        	            	-  }
        	            	-}
        	            	+File [OUTPUT_DIR]/out.requests.txt not found
        	            	 
        	            	+Exit code: 1
        	            	+
        	Test:       	TestAccept/pipelines/test/not-enabled
        	Messages:   	pipelines/test/not-enabled/output.txt vs /tmp/TestAcceptpipelinestestnot-enabled670331469/002/output.txt
    acceptance_test.go:1355: Missing output file: out.not-enabled.xml
TestAccept/pipelines/test/run-control/timeout: branch ✅ | main (73a2b68) ❌ | latest (v1.26.0) ❌

main (73a2b68):

acceptance_test.go:1384: 
        	Error Trace:	/home/runner/work/cli/cli/libs/testdiff/testdiff.go:22
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1384
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1125
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:670
        	Error:      	Not equal: 
        	            	expected: "\n=== --timeout stops waiting but not the update, and prints the command that reads the results\n\n>>> musterr [CLI] pipelines test --pipeline-id [UUID][0] --resume [UUID][1] --timeout 3s --junit-xml out.report.xml\nPipeline: [UUID][0]\nUpdate: [UUID][1]\nRunning tests...\nJUnit report: out.report.xml\nError: stopped waiting for update [UUID][1] after 3s; the update keeps running, to get its results run: databricks pipelines test --pipeline-id [UUID][0] --junit-xml out.report.xml --resume [UUID][1]\n"
        	            	actual  : "\n=== --timeout stops waiting but not the update, and prints the command that reads the results\n\n>>> musterr [CLI] pipelines test --pipeline-id [UUID][0] --resume [UUID][1] --timeout 3s --junit-xml out.report.xml\nError: pipelines test is experimental. Set DATABRICKS_CLI_PIPELINES_TEST=true to enable it\n"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -4,7 +4,3 @@
        	            	 >>> musterr [CLI] pipelines test --pipeline-id [UUID][0] --resume [UUID][1] --timeout 3s --junit-xml out.report.xml
        	            	-Pipeline: [UUID][0]
        	            	-Update: [UUID][1]
        	            	-Running tests...
        	            	-JUnit report: out.report.xml
        	            	-Error: stopped waiting for update [UUID][1] after 3s; the update keeps running, to get its results run: databricks pipelines test --pipeline-id [UUID][0] --junit-xml out.report.xml --resume [UUID][1]
        	            	+Error: pipelines test is experimental. Set DATABRICKS_CLI_PIPELINES_TEST=true to enable it
        	            	 
        	Test:       	TestAccept/pipelines/test/run-control/timeout
        	Messages:   	pipelines/test/run-control/timeout/output.txt vs /tmp/TestAcceptpipelinestestrun-controltimeout379938245/002/output.txt
    acceptance_test.go:1384: 
        	Error Trace:	/home/runner/work/cli/cli/libs/testdiff/testdiff.go:22
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1384
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1137
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:670
        	Error:      	Not equal: 
        	            	expected: "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<testsuites>\n  <testsuite name=\"[UUID][0]\" tests=\"2\" failures=\"0\" errors=\"2\" skipped=\"0\">\n    <testcase classname=\"tests.test_orders\" name=\"test_slow\">\n      <error message=\"no result received\"></error>\n    </testcase>\n    <testcase name=\"pipelines test\">\n      <error message=\"stopped waiting for update [UUID][1] after 3s; the update keeps running, to get its results run: databricks pipelines test --pipeline-id [UUID][0] --junit-xml out.report.xml --resume [UUID][1]\">stopped waiting for update [UUID][1] after 3s; the update keeps running, to get its results run: databricks pipelines test --pipeline-id [UUID][0] --junit-xml out.report.xml --resume [UUID][1]&#xA;Update: [UUID][1]</error>\n    </testcase>\n  </testsuite>\n</testsuites>\n"
        	            	actual  : "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<testsuites>\n  <testsuite name=\"[UUID]\" tests=\"1\" failures=\"0\" errors=\"1\" skipped=\"0\">\n    <testcase name=\"pipelines test\">\n      <error message=\"pipelines test is experimental. Set DATABRICKS_CLI_PIPELINES_TEST=true to enable it\">pipelines test is experimental. Set DATABRICKS_CLI_PIPELINES_TEST=true to enable it</error>\n    </testcase>\n  </testsuite>\n</testsuites>\n"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -2,8 +2,5 @@
        	            	 <testsuites>
        	            	-  <testsuite name="[UUID][0]" tests="2" failures="0" errors="2" skipped="0">
        	            	-    <testcase classname="tests.test_orders" name="test_slow">
        	            	-      <error message="no result received"></error>
        	            	-    </testcase>
        	            	+  <testsuite name="[UUID]" tests="1" failures="0" errors="1" skipped="0">
        	            	     <testcase name="pipelines test">
        	            	-      <error message="stopped waiting for update [UUID][1] after 3s; the update keeps running, to get its results run: databricks pipelines test --pipeline-id [UUID][0] --junit-xml out.report.xml --resume [UUID][1]">stopped waiting for update [UUID][1] after 3s; the update keeps running, to get its results run: databricks pipelines test --pipeline-id [UUID][0] --junit-xml out.report.xml --resume [UUID][1]&#xA;Update: [UUID][1]</error>
        	            	+      <error message="pipelines test is experimental. Set DATABRICKS_CLI_PIPELINES_TEST=true to enable it">pipelines test is experimental. Set DATABRICKS_CLI_PIPELINES_TEST=true to enable it</error>
        	            	     </testcase>
        	Test:       	TestAccept/pipelines/test/run-control/timeout
        	Messages:   	pipelines/test/run-control/timeout/out.report.xml vs /tmp/TestAcceptpipelinestestrun-controltimeout379938245/001/out.report.xml

latest (v1.26.0):

acceptance_test.go:1384: 
        	Error Trace:	/home/runner/work/cli/cli/libs/testdiff/testdiff.go:22
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1384
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1125
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:670
        	Error:      	Not equal: 
        	            	expected: "\n=== --timeout stops waiting but not the update, and prints the command that reads the results\n\n>>> musterr [CLI] pipelines test --pipeline-id [UUID][0] --resume [UUID][1] --timeout 3s --junit-xml out.report.xml\nPipeline: [UUID][0]\nUpdate: [UUID][1]\nRunning tests...\nJUnit report: out.report.xml\nError: stopped waiting for update [UUID][1] after 3s; the update keeps running, to get its results run: databricks pipelines test --pipeline-id [UUID][0] --junit-xml out.report.xml --resume [UUID][1]\n"
        	            	actual  : "\n=== --timeout stops waiting but not the update, and prints the command that reads the results\n\n>>> musterr [CLI] pipelines test --pipeline-id [UUID][0] --resume [UUID][1] --timeout 3s --junit-xml out.report.xml\nError: unknown flag: --resume\n\nUsage:\n  databricks pipelines test [flags] [KEY] [TEST_SELECTION...]\n\nPipeline Test Flags:\n      --pipeline-id string       ID of the pipeline to test, instead of a KEY.\n      --pytest-arg stringArray   Argument to pass to pytest (repeatable). Only -k and -m are supported.\n\nFlags:\n  -h, --help   help for test\n\nGlobal Flags:\n      --debug            enable debug logging\n  -o, --output type      output type: text or json (default text)\n  -p, --profile string   ~/.databrickscfg profile\n  -t, --target string    bundle target to use (if applicable)\n      --var strings      set values for variables defined in project config. Example: --var=\"foo=bar\"\n\n"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -4,7 +4,21 @@
        	            	 >>> musterr [CLI] pipelines test --pipeline-id [UUID][0] --resume [UUID][1] --timeout 3s --junit-xml out.report.xml
        	            	-Pipeline: [UUID][0]
        	            	-Update: [UUID][1]
        	            	-Running tests...
        	            	-JUnit report: out.report.xml
        	            	-Error: stopped waiting for update [UUID][1] after 3s; the update keeps running, to get its results run: databricks pipelines test --pipeline-id [UUID][0] --junit-xml out.report.xml --resume [UUID][1]
        	            	+Error: unknown flag: --resume
        	            	 
        	            	+Usage:
        	            	+  databricks pipelines test [flags] [KEY] [TEST_SELECTION...]
        	            	+
        	            	+Pipeline Test Flags:
        	            	+      --pipeline-id string       ID of the pipeline to test, instead of a KEY.
        	            	+      --pytest-arg stringArray   Argument to pass to pytest (repeatable). Only -k and -m are supported.
        	            	+
        	            	+Flags:
        	            	+  -h, --help   help for test
        	            	+
        	            	+Global Flags:
        	            	+      --debug            enable debug logging
        	            	+  -o, --output type      output type: text or json (default text)
        	            	+  -p, --profile string   ~/.databrickscfg profile
        	            	+  -t, --target string    bundle target to use (if applicable)
        	            	+      --var strings      set values for variables defined in project config. Example: --var="foo=bar"
        	            	+
        	            	+
        	Test:       	TestAccept/pipelines/test/run-control/timeout
        	Messages:   	pipelines/test/run-control/timeout/output.txt vs /tmp/TestAcceptpipelinestestrun-controltimeout2923980557/002/output.txt
    acceptance_test.go:1355: Missing output file: out.report.xml

2 more failure details omitted, see the full report in the run.

@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Collaborator

❌ Integration test report

Commit: 453fa56

Run: 43092531796

@tharunbhuk-db
tharunbhuk-db force-pushed the tharun/pipelines-test-j1-testing-sources branch from ed5750c to 75f2713 Compare November 10, 2026 01:16
@tharunbhuk-db
tharunbhuk-db changed the base branch from tharun/pipelines-test-j1-base to tharun/pipelines-test-b3-dependency-checks November 10, 2026 01:16
@tharunbhuk-db
tharunbhuk-db force-pushed the tharun/pipelines-test-j1-testing-sources branch from 75f2713 to ccdd807 Compare November 10, 2026 10:07
@tharunbhuk-db
tharunbhuk-db changed the base branch from tharun/pipelines-test-b3-dependency-checks to tharun/pipelines-test-b4-testing-deploy November 10, 2026 10:07
@tharunbhuk-db
tharunbhuk-db force-pushed the tharun/pipelines-test-j1-testing-sources branch 2 times, most recently from 2e9b82e to a149708 Compare November 10, 2026 11:11
@tharunbhuk-db
tharunbhuk-db force-pushed the tharun/pipelines-test-j1-testing-sources branch 2 times, most recently from 490849a to 6c31046 Compare November 10, 2026 17:14
@tharunbhuk-db
tharunbhuk-db force-pushed the tharun/pipelines-test-j1-testing-sources branch from 6c31046 to bbb4adb Compare November 10, 2026 17:22
@tharunbhuk-db
tharunbhuk-db marked this pull request as ready for review November 10, 2026 18:11
@tharunbhuk-db
tharunbhuk-db requested review from a team as code owners November 10, 2026 18:11
@tharunbhuk-db
tharunbhuk-db removed this pull request from stack #7755 November 18, 2026 09:28
@tharunbhuk-db
tharunbhuk-db force-pushed the tharun/pipelines-test-j1-testing-sources branch from 95e39e8 to 3a6c0e5 Compare November 18, 2026 09:29
@tharunbhuk-db
tharunbhuk-db changed the base branch from tharun/pipelines-test-changelog to tharun/pipelines-test-b3-testing-paths November 18, 2026 09:31
@tharunbhuk-db
tharunbhuk-db removed this pull request from stack #7758 November 18, 2026 10:10
@tharunbhuk-db
tharunbhuk-db force-pushed the tharun/pipelines-test-j1-testing-sources branch from 8b33ae8 to 6983923 Compare November 18, 2026 10:10
@tharunbhuk-db
tharunbhuk-db force-pushed the tharun/pipelines-test-j1-testing-sources branch from 6983923 to 5986de8 Compare November 18, 2026 10:23
@tharunbhuk-db

Copy link
Copy Markdown
Contributor Author

@pietern thanks for the review. Reworked the stack, ready for another look: it is now 3 PRs (#7427 → #7437 → #7429).

@tharunbhuk-db
tharunbhuk-db added this pull request to the merge queue Nov 18, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to invalid changes in the merge commit Nov 18, 2026
Base automatically changed from tharun/pipelines-test-b3-testing-paths to main November 18, 2026 12:10
tharunbhuk-db and others added 7 commits November 18, 2026 04:10
The pipelines test acceptance bundles declare testing, as the service requires
testing.sources for a test-only update.

When Start Update fails with PIPELINE_TEST_NO_TEST_SOURCES and the bundle sets
testing for the pipeline, the error says to deploy first: the bundle changed
after its last deploy.

Add a live acceptance test that deploys a bundle with testing, including a local
test dependency after -e, and runs its tests. Locally it uses stubs for the
test run. Against a real workspace it runs only when
TEST_PIPELINES_TESTING_ENABLED is set.

Co-authored-by: Isaac <no-reply@databricks.com>
The command no longer needs DATABRICKS_CLI_PIPELINES_TEST=true. Whether a
workspace can run pipeline tests is decided by the service, so a run of all
tests now always sends test_options, empty when nothing is selected. Where the
testing API is off, the service rejects such a request with a clear error,
instead of running its older test-only update.

Co-authored-by: Isaac <no-reply@databricks.com>
The command no longer needs an environment variable to run.

Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
@tharunbhuk-db
tharunbhuk-db force-pushed the tharun/pipelines-test-j1-testing-sources branch from 5986de8 to 7ca3bea Compare November 18, 2026 12:10
@tharunbhuk-db

Copy link
Copy Markdown
Contributor Author

/merge

@tharunbhuk-db
tharunbhuk-db added this pull request to the merge queue Nov 18, 2026
Merged via the queue into main with commit 76bf50d Nov 18, 2026
25 checks passed
@tharunbhuk-db
tharunbhuk-db deleted the tharun/pipelines-test-j1-testing-sources branch November 18, 2026 12:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants